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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1]A mixed reality presentation system which carries out the superimposed display of the 
virtual body to real space, comprising: 

A mixed reality presenting means which carries out the superimposed display of said virtual body 
seen from said subjectivity view position of said player to said real space seen from a 
subjectivity view position of a player. 

An object viewpoint image imaging means which picturizes an object viewpoint image of said real 
space seen from an object view position. 

An object viewpoint video generating means which generates an object viewpoint image of said 
virtual body seen from said object view position. 

An object viewpoint video compositing means which compounds a mixed reality image seen from 
said object viewpoint from an object viewpoint image of said real space, and an object viewpoint 
image of said virtual body. 

[Claim 2]The mixed reality presentation system comprising according to claim 1: 
A subjectivity viewpoint image imaging means in which said mixed reality presenting means 
picturizes further a subjectivity viewpoint image of said real space seen from said subjectivity 
view position of said player. 

A subjectivity viewpoint video generating means which generates a subjectivity viewpoint image 
of said virtual body seen from said subjectivity view position of said player. 
A subjectivity viewpoint video compositing means which compounds a mixed reality image seen 
from said subjectivity viewpoint from a subjectivity viewpoint image of said real space, and a 
subjectivity viewpoint image of said virtual body. 

A presenting means which shows said player a mixed reality image seen from said subjectivity 
view position. 

[Claim 3]The mixed reality presentation system comprising according to claim 1: 
A subjectivity viewpoint video generating means in which said mixed reality presenting means 
generates further a subjectivity viewpoint image of said virtual body seen from said subjectivity 
view position of said player. 

A presenting means which penetrates said real space and shows said player a subjectivity 
viewpoint image of said virtual body on a display surface which can be viewed. 

[Claim 4]The mixed reality presentation system according to claim 1 to 3, wherein it has further 
an information creating means which generates information about drawing of said virtual body 
and both said object viewpoint video generating means and said subjectivity viewpoint video 
generating means generate an image of said virtual body using information about drawing of said 
virtual body. 

[Claim 5]The mixed reality presentation system according to claim 4, wherein said information 
creating means generates information on appearance of said virtual body, and information on a 
positional attitude of a virtual body as information about drawing of said virtual body. 
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[Claim 6]The mixed reality presentation system according to claim 1 or 4 which a parameter of 
said object viewpoint image imaging means is known, and is characterized by said object 
viewpoint video generating means generating an object viewpoint image of said virtual body 
according to said known parameter 

[Claim 7]A part of parameter of said object viewpoint image imaging means is variable, have 
further a measurement means which measures change of said parameter, and said object 
viewpoint video generating means, The mixed reality presentation system according to claim 1 or 
4 generating an object viewpoint image of said virtual body according to said parameter 
measured by said measurement means. 

[Claim 8]The mixed reality presentation system according to claim 7 t wherein said parameter of 
said object viewpoint image imaging means which said measurement means measures contains 
one of a positional attitude of a viewpoint, and rates of a zoom ratio at least. 
[Claim 9]When two or more said object viewpoint image imaging means exist, an object viewpoint 
image of two or more of said real space is inputted from said two or more object viewpoint image 
imaging means, It has further a selecting means which outputs an object viewpoint image of said 
real space inputted from said one selected object viewpoint image imaging means to said object 
viewpoint video compositing means, The mixed reality presentation system according to claim 1 , 
wherein said object viewpoint video compositing means generates an object viewpoint image of 
said virtual body using a parameter of said object viewpoint image imaging means which said 
selecting means chose. 

[Claim 10]A mixed reality presenting method which carries out the superimposed display of the 
virtual body to real space, comprising: 

A mixed reality presentation process of carrying out the superimposed display of said virtual 
body seen from said subjectivity view position of said player to said real space seen from a 
subjectivity view position of a player. 

An object viewpoint image imaging step which picturizes an object viewpoint image of said real 
space seen from an object view position. 

An object viewpoint image generation process of generating an object viewpoint image of said 
virtual body seen from said object view position. 

An object viewpoint picture composite process of compounding a mixed reality image seen from 
said object viewpoint from an object viewpoint image of said real space, and an object viewpoint 
image of said virtual body. 

[Claim 1 1]The mixed reality presenting method comprising according to claim 10: 
A subjectivity viewpoint image imaging step in which said mixed reality presentation process 
picturizes further a subjectivity viewpoint image of said real space seen from said subjectivity 
view position of said player, 

A subjectivity viewpoint image generation process of generating a subjectivity viewpoint image of 
said virtual body seen from said subjectivity view position of said player. 

A subjectivity viewpoint picture composite process of compounding a mixed reality image seen 
from said subjectivity viewpoint from a subjectivity viewpoint image of said real space, and a 
subjectivity viewpoint image of said virtual body, 

A presentation process of showing said player a mixed reality image seen from said subjectivity 
view position. 

[Claim 12]The mixed reality presenting method comprising according to claim 10: 
A subjectivity viewpoint image generation process that said mixed reality presentation process 
generates further a subjectivity viewpoint image of said virtual body seen from said subjectivity 
view position of said player. 

A presentation process of penetrating said real space and showing said player a subjectivity 
viewpoint image of said virtual body on a display surface which can be viewed. 

[Claim 13]Have further the information generation process of generating information about 
drawing of said virtual body, and in said object viewpoint image generation process and said 
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subjectivity viewpoint image generation process. The mixed reality presenting method according 
to claim 10 to 12 generating an image of said virtual body using information about both drawing 
of said virtual body. 

[Claim 14]The mixed reality presenting method according to claim 13 characterized by generating 
information on appearance of said virtual body, and information on a positional attitude of a 
virtual body as information about drawing of said virtual body at said information generation 
process. 

[Claim 15]The mixed reality presenting method according to claim 10 or 13 which a parameter of 
a means to perform an object viewpoint image image pick-up is known, and is characterized by 
generating an object viewpoint image of said virtual body according to said known parameter in 
said object viewpoint image generation process. 

[Ciaim 16]A part of parameter of a means to perform an object viewpoint image image pick-up is 
variable, have further a measurement step which measures change of said parameter, and said 
object viewpoint image generation process, The mixed reality presenting method according to 
claim 10 or 13 generating an object viewpoint image of said virtual body according to said 
parameter measured in said measurement step. 

[Claim 17]The mixed reality presenting method according to claim 16, wherein said parameter of 
a means to perform an object viewpoint image image pick-up measured in said measurement 
step contains one of a positional attitude of a viewpoint, and rates of a zoom ratio at least. 
[Claim 18]When two or more means to perform said object viewpoint image image pick-up exist, 
an object viewpoint image of two or more of said real space is inputted from a means to perform 
said two or more object viewpoint image image pick-ups, It has further a selection process which 
outputs an object viewpoint image of said real space inputted from a means to perform said one 
selected object viewpoint image image pick-up to a means to perform said object viewpoint 
picture composite, The mixed reality presenting method according to claim 10 generating an 
object viewpoint image of said virtual body using a parameter of a means to perform said object 
viewpoint image image pick-up selected in said selection process in said object viewpoint picture 
composite process. 

[Claim 19]A storage which stores a program code which carries out the superimposed display of 
the virtual body to real space by making it read into a computer characterized by comprising the 
following. 

A program code of a mixed reality presentation process of carrying out the superimposed display 
of said virtual body seen from said subjectivity view position of said player to said real space 
seen from a subjectivity view position of a player. 

A program code of an object viewpoint image imaging step which picturizes an object viewpoint 
image of said real space seen from an object view position. 

A program code of an object viewpoint image generation process of generating an object 
viewpoint image of said virtual body seen from said object view position. 
A program code of an object viewpoint picture composite process of compounding a mixed 
reality image seen from said object viewpoint from an object viewpoint image of said real space, 
and an object viewpoint image of said virtual body. 

[Claim 20]A storage which stores the program code according to claim 19, comprising: 
A program code of a subjectivity viewpoint image imaging step with which a program code of said 
mixed reality presentation process picturizes further a subjectivity viewpoint image of said real 
space seen from said subjectivity view position of said player. 

A program code of a subjectivity viewpoint image generation process of generating a subjectivity 
viewpoint image of said virtual body seen from said subjectivity view position of said player. 
A program code of a subjectivity viewpoint picture composite process of compounding a mixed 
reality image seen from said subjectivity viewpoint from a subjectivity viewpoint image of said 
real space, and a subjectivity viewpoint image of said virtual body. 

A program code of a presentation process of showing said player a mixed reality image seen 
from said subjectivity view position. 
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[Claim 21]A storage which stores the program code according to ciaim 19 f comprising: 
A program code of a subjectivity viewpoint image generation process that a program code of said 
mixed reality presentation process generates further a subjectivity viewpoint image of said 
virtual body seen from said subjectivity view position of said player. 

A program code of a presentation process of penetrating said real space and showing said player 
a subjectivity viewpoint image of said virtual body on a display surface which can be viewed. 

[Claim 22]It has further a program code of an information generation process of generating 
information about drawing of said virtual body, A storage which stores the program code 
according to claim 19 to 21, wherein both a program code of said object viewpoint image 
generation process and a program code of said subjectivity viewpoint image generation process 
generate an image of said virtual body using information about drawing of said virtual body. 
[Claim 23]A storage which stores the program code according to claim 22, wherein a program 
code of said information generation process generates information on appearance of said virtual 
body, and information on a positional attitude of a virtual body as information about drawing of 
said virtual body. 

[Claim 24]A storage which a parameter of said object viewpoint image imaging means is known, 
and stores the program code according to claim 19 or 21, wherein a program code of said object 
viewpoint image generation process generates an object viewpoint image of said virtual body 
according to said known parameter 

[Claim 25]A part of parameter of a means to perform said object viewpoint image image pick-up 
is variable, have further a measurement means which measures change of said parameter, and a 
program code of said object viewpoint image generation process, A storage which stores the 
program code according to claim 19 or 22 generating an object viewpoint image of said virtual 
body according to said parameter measured by said measurement means. 
[Claim 26]A storage which stores the program code according to claim 25, wherein said 
parameter of a means to perform an object viewpoint image image pick-up which said 
measurement means measures contains one of a positional attitude of a viewpoint, and rates of 
a zoom ratio at least 

[Claim 27]When two or more means to perform said object viewpoint image image pick-up exist, 
an object viewpoint image of two or more of said real space is inputted from a means to perform 
said two or more object viewpoint image image pick-ups, It has further a program code of a 
selection process which outputs an object viewpoint image of said real space inputted from a 
means to perform said one selected object viewpoint image image pick-up to a means to perform 
said object viewpoint picture composite, A storage which stores the program code according to 
claim 19, wherein a program code of said object viewpoint picture composite process generates 
an object viewpoint image of said virtual body using a parameter of a means to perform said 
object viewpoint image image pick-up which a program code of said selection process chose. 



[Translation done.] 
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[Detailed Description of the Invention] 
[0001] 

[Field of the Invention]This invention relates to the mixed reality presentation system, mixed 
reality presenting method, and storage which carry out the superimposed display of the virtual 
body to real space. 

[0002] Especially a player is related with a mixed reality presentation system, a mixed reality 
presenting method, and a storage with the function which generates the mixed reality space 
currently observed from the subjectivity viewpoint as an image seen from the object viewpoint 
[0003] 

[Description of the Prior Art]AR game device with which the conventional mixed reality (following 
AR) game system controls AR game, It is ** constituted with the head mount display 
(henceforth, HMD) with which the player which performs AR game equips, and the display which 
shows third parties other than a player the subjectivity viewpoint image of said player. 
[0004]AR game device is generating the information about drawing of a virtual body at the same 
time it controls AR game. AR game device generates the synthetic video which is an image 
which compounded the image of the virtual body generated using the information about drawing 
of this virtual body, and the image of the actual space in the subjectivity viewpoint of the player 
picturized with the camera equipped with or built in HMD, And this synthetic video is displayed 
on the display screen of HMD, The player is performing AR game, looking at this synthetic video. 
The third party who observes this AR game besides a player is shown by displaying the image of 
the subjectivity viewpoint of a player which is this synthetic video on the display prepared to 
third parties other than a player. 

[0005]In the program of some televisions, the virtual studio which constitutes the set of the 
stage of a program with a virtual body is used. The image of the program in this virtual studio is 
generated by compounding the on-the-spot photo image of the characters of the program 
picturized with the set-up camera, and the position of this camera and the image of the virtual 
studio which is visible with a posture. 
[0006] 

[Problem(s) to be Solved by the Invention]In the conventional AR (mixed reality) game system, 
third parties (spectator) other than a player can see only the image (mainly subjectivity 
viewpoint of a player) generated for the player. Therefore, images of AR game by the viewpoint 
from other than the subjectivity viewpoint of a player, such as an overview of AR game seen 
from the object viewpoint and an image of AR game from the viewpoint of a spectators choice, 
could not be seen, but neither grasp of the situation of the whole AR game nor watching a game 
of AR game from the viewpoint of a spectator's choice was completed, 
[0007]The image of the program using a virtual studio is not able to be based on the object 
viewpoint image of third parties other than the characters of a program, and was not able to 
generate the image by the subjectivity viewpoint of the characters of a program. 
[0008]Therefore, it aims at generating the image of AR game by the subjectivity viewpoint of a 
player, and generating the overview of AR game seen from the object viewpoint, and the image of 
AR game from the viewpoint of a spectator's choice in this invention. 
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[0009] 

[Means for Solving the Problem]In order to attain the purpose of this invention, the mixed reality 
presentation system according to claim 1 which is a suitable embodiment of this invention is 
provided with the following composition. Namely, a mixed reality presenting means which is a 
mixed reality presentation system which carries out the superimposed display of the virtual body 
to real space, and carries out the superimposed display of said virtual body seen from said 
subjectivity view position of said player to said real space seen from a subjectivity view position 
of a player, An object viewpoint image imaging means which picturizes an object viewpoint image 
of said real space seen from an object view position, It has an object viewpoint video 
compositing means which compounds a mixed reality image seen from said object viewpoint from 
an object viewpoint video generating means which generates an object viewpoint image of said 
virtual body seen from said object view position, an object viewpoint image of said real space, 
and an object viewpoint image of said virtual body. 

[0010]As a result an image which looked at signs that a player was operating a virtual body in 
mixed reality space, from an object view position can be generated, and third parties other than a 
player can be shown. 

[001 1]A mixed reality presentation system which is a suitable embodiment of this invention is 
provided with the feature of the following according to claim 2. Namely, a subjectivity viewpoint 
image imaging means in which said mixed reality presenting means picturizes further a 
subjectivity viewpoint image of said real space seen from said subjectivity view position of said 
player, A subjectivity viewpoint video generating means which generates a subjectivity viewpoint 
image of said virtual body seen from said subjectivity view position of said player, It has a 
subjectivity viewpoint video compositing means which compounds a mixed reality image seen 
from said subjectivity viewpoint from a subjectivity viewpoint image of said real space, and a 
subjectivity viewpoint image of said virtual body, and a presenting means which shows said player 
a mixed reality image seen from said subjectivity view position, 

[001 2]A mixed reality presentation system which is a suitable embodiment of this invention is 
provided with the feature of the following according to claim 3. Namely, in [ said mixed reality 
presenting means penetrates a subjectivity viewpoint video generating means which generates 
further a subjectivity viewpoint image of said virtual body seen from said subjectivity view 
position of said player, and said real space, and ] a display surface top which can be viewed, It 
has a presenting means which shows said player a subjectivity viewpoint image of said virtual 
body. 

[001 3]A mixed reality presentation system which is a suitable embodiment of this invention is 
provided with the feature of the following according to claim 4. That is, it has further an 
information creating means which generates information about drawing of said virtual body, and 
both said object viewpoint video generating means and said subjectivity viewpoint video 
generating means generate an image of said virtual body using information about drawing of said 
virtual body. 

[001 4]A mixed reality presentation system which is a suitable embodiment of this invention is 
provided with the feature of the following according to claim 5. That is, said information creating 
means generates information on appearance of said virtual body, and information on a positional 
attitude of a virtual body as information about drawing of said virtual body. 
[001 5]A mixed reality presentation system which is a suitable embodiment of this invention is 
provided with the feature of the following according to claim 6. That is, a parameter of said 
object viewpoint image imaging means is known, and said object viewpoint video generating 
means generates an object viewpoint image of said virtual body according to said known 
parameter. 

[001 6]A mixed reality presentation system which is a suitable embodiment of this invention is 
provided with the feature of the following according to claim 7. Namely, a part of parameter of 
said object viewpoint image imaging means is variable, have further a measurement means which 
measures change of said parameter, and said object viewpoint video generating means, 
According to said parameter measured by said measurement means, an object viewpoint image 
of said virtual body is generated. 
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[0017]When a parameter of an object viewpoint image imaging means is changed, an object 
viewpoint video generating means inputs a parameter from a measurement means, and generates 
an object viewpoint image according to it. 

[001 8]A mixed reality presentation system which is a suitable embodiment of this invention is 
provided with the feature of the following according to claim 8. That is, said parameter of said 
object viewpoint image imaging means which said measurement means measures contains one of 
a positional attitude of a viewpoint, and rates of a zoom ratio at least 

[001 9] An object viewpoint image of a virtual body is generated according to a camera parameter 
(an external parameter (positional attitude of a viewpoint), and an internal parameter (a rate of a 
zoom ratio, an aspect ratio, an optical axis center position, a distortion rate)) of an object 
viewpoint image imaging means. It is preferred for a camera parameter which a measurement 
means measures that all things dynamically changed among these parameters are included. 
[0020] 

[Embodiment of the lnvention]According to an accompanying drawing, the suitable embodiment 
which applied this invention is described in detail below, 

[0021][A 1st embodiment] This embodiment shows the mixed reality presentation system and 
mixed reality presenting method which show third parties (the following, spectator) other than a 
player signs that the player is performing AR game in AR (mixed reality) space as one of the 
suitable embodiments of this invention. 
[0022]This embodiment is shown in drawin g 1, 

[0023] 101 is the player which is performing AR (mixed reality) game, and has equipped the head 
with the head mount display (following, HMD) 107. HMD107 in this embodiment is a video see- 
through type. About HMD of a bidet see-through type, explanation here is omitted for publicly 
known art. 

[0024] 102 is a virtual body and is a character which appears in AR game. This virtual body 102 is 
a three-dimensional virtual body constituted by the polygon. The virtual body 102 is generated 
by the method of mentioning later with the AR game device 104. And the player 101 can see the 
image (henceforth, AR image) superimposed on the virtual body 102 by real space from a 
subjectivity viewpoint by equipping a head with HMD107, and, as a result, can perform AR game. 
The component of the virtual body 102 is not having restricted to the polygon. Otherwise, it may 
be constituted by spatial light beam data. 

[0025] 103 is the object viewpoint image imaging camera (the following, camera) 103, and is a 
camera which picturizes signs that the player 101 is performing AR game. This camera 103 is 
being fixed to the position and the posture in this embodiment. The image (following and object 
viewpoint on-the-spot photo image) picturized with the camera 103 is sent to the AR game 
device 104 via the cable in a figure. The picturized image is an on-the-spot photo image, and the 
virtual body 102 is not picturized by this on-the-spot photo image, 

[0026] 104 is AR game device and is generating AR image outputted to control of AR game, 
generation of the image of the virtual body 102, HMD107, and the display 106. Here, the image 
which the AR game device 104 outputs to HMD 107 is an AR image (following and subjectivity 
viewpoint AR image) which compounded the on-the-spot photo image (following and subjectivity 
viewpoint on-the-spot photo image) of the real space inputted from HMD107, and the image 
(following and subjectivity viewpoint virtual image) of the virtual body 102 seen from the 
subjectivity viewpoint. On the other hand, the image which the AR game device 104 outputs to 
the display 106 is an AR image (following and object viewpoint AR image) which compounded the 
image (following and object viewpoint virtual image) of the virtual body 102 seen from the object 
viewpoint and the object viewpoint on-the-spot photo image picturized with the camera 103. 
[0027]105 is a stand of the realistic object used in AR game. 

[0028] 106 is a display, is displaying the object viewpoint AR image which got with above- 
mentioned one and was generated by the AR game device 104, and shows third parties other 
than player 101 signs that the player 101 is performing AR game. 

[0029] 107 is above-mentioned HMD and displays the subjectivity viewpoint AR image generated 
by the AR game device 104 on the display screen 501 shown in drawi ng 5 of HMD 107. HMD 107 
picturizes the subjectivity viewpoint on-the-spot photo image seen from the subjectivity 
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viewpoint of the player 101. The picturized image is sent to the AR game device 104 via the 
cable in a figure. 

[0030]It explains using drawi ng 2 in which the flow of this processing was shown about flowing 
into processing based on the above composition until the AR game device 104 displayed a 
subjectivity viewpoint AR image and an object viewpoint AR image on HMD107 and the display 
106, respectively. 

[0031]201 is the game condition Management Department built in the AR game device 104 f and 
manages the states (the information about drawing of the virtual body 102, the score of the 
player 101, AR game round number, etc.) of AR game. Here, the information about drawing of the 
virtual body 102 is constituted by the information on the positional attitude of the virtual body 
102 in the information and world coordinate system of appearance of the virtual body 102. 
[0032]The information on the appearance of the virtual body 102 is information about the 
polygon which constitutes the virtual body 102, and is the number of polygons, a coordinate 
value of each polygon, a color of each polygon, etc. When texture mapping is given to the virtual 
body 102, the size of a texture, the file name of a texture, etc, are contained in the information 
on the appearance of the virtual body 102. 

[0033]204 is a camera parameter instrumentation department and is performing 
measurement/management of the camera parameter which is a parameter of HMD107 and the 
camera 103. The position of a viewpoint and the information on a posture which are external 
parameters, and the information, including a field angle, a focal distance, distortion, etc., which is 
interna! parameters are included in the camera parameter which the camera parameter 
instrumentation department 204 manages. The camera parameter instrumentation department 
204 controls the sensor not shown in the figure of HMD 107 at the same time it holds the internal 
parameter of HMD107 as known information, The external parameter (the position of a viewpoint 
and the information on a posture) of HMD107 was measured, and the camera parameter of 
HMD 107 is managed. The camera parameter instrumentation department 204 has managed the 
camera parameter of the camera 103 as known information. 

[0034]202 is the object viewpoint video generation part built in the AR game device 104 t and 

generates an object viewpoint virtual image based on the information about drawing of the virtual 

body 102 inputted from the game condition Management Department 201. 

[0035]203 is the object viewpoint image compositing section built in the AR game device 104, 

compounds the object viewpoint virtual image generated by the object viewpoint video 

generation part 202 and the object viewpoint on-the-spot photo image inputted from the camera 

1 03, and generates an object viewpoint AR image. 

[0036]First, generation of a subjectivity viewpoint image is explained, 

[0037]The game condition Management Department 201 updates the information about drawing 
of the virtual body 102 if needed, and outputs update information to the subjectivity viewpoint 
video generation part 212. The subjectivity viewpoint video generation part 212 generates the 
image of the virtual body 102 based on the camera parameter of HMD107 inputted as the 
information about drawing of the virtual body 102 inputted from the game condition Management 
Department 201 from the camera parameter instrumentation department 204. Specifically, the 
coordinate transformation matrix M1 from object frame to the image-coordinates system of 
HMD107 is computed based on the method of mentioning later from the information on the 
positional attitude of the virtual body 102, and the camera parameter of HMD107, By hanging the 
coordinate transformation matrix M1 computed to the apex-coordinates value in the object 
frame of each polygon which constitutes the virtual body 102. The image-coordinates value of 
the peak of each polygon which constitutes the virtual body 102 is computed, and the image 
(subjectivity viewpoint virtual image) of the virtual body 102 seen from the view position posture 
of HMD107 is generated. And the generated subjectivity viewpoint virtual image is outputted to 
the subjectivity viewpoint image compositing section 213. 

[0038]The subjectivity viewpoint on-the-spot photo image which the player 101 looks at via 
HMD107 is inputted into the subjectivity viewpoint image compositing section 213 from HMD107. 
And the subjectivity viewpoint image compositing section 213 generates the subjectivity 
viewpoint AR image which is synthetic video with the subjectivity viewpoint virtual image 
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inputted from this subjectivity viewpoint on-the-spot photo image and the subjectivity viewpoint 
video generation part 212, and outputs this subjectivity viewpoint AR image to HMD107. A 
subjectivity viewpoint AR image is displayed on the display screen 501 of HMD107, and the 
player 101 performs AR game, seeing this subjectivity viewpoint AR image. 
[0039]Next generation of an object viewpoint image is explained. 

[0040]The game condition Management Department 201 updates the information about drawing 
of the virtual body 102 if needed, and outputs update information to the object viewpoint video 
generation part 202. 

[0041]Since it is the processing same about generation of the image of the virtual body 102 as 
the processing in generation of a subjectivity viewpoint image, explanation is omitted here. 
[0042]The object viewpoint video generation part 202 generates the image of the virtual body 
102 based on the camera parameter of the camera 103 inputted as the information about 
drawing of the virtual body 102 inputted from the game condition Management Department 201 
from the camera parameter instrumentation department 204. Specifically, the coordinate- 
system-transformation procession M2 from object frame to the image-coordinates system of 
the camera 103 is computed based on the method of mentioning later from the information on 
the positional attitude of the virtual body 102, and the camera parameter of the camera 103, By 
hanging the coordinate transformation matrix M2 computed to the apex-coordinates value in the 
object frame of each polygon which constitutes the virtual body 102. The image-coordinates 
value of the peak of each polygon which constitutes the virtual body 102 is computed, and the 
image (object viewpoint virtual image) of the virtual body 103 seen from the view position 
posture of the camera 103 is generated. And the generated object viewpoint virtual image is 
outputted to the object viewpoint image compositing section 203. 

[0043]An object viewpoint on-the-spot photo image is inputted into the object viewpoint image 
compositing section 203 from the camera 103. And the object viewpoint image compositing 
section 203 generates the object viewpoint AR image which is synthetic video with the object 
viewpoint virtual image inputted from this object viewpoint onH:he-spot photo image and the 
object viewpoint video generation part 202 T and outputs this object viewpoint AR image to the 
display 106. This object viewpoint AR image is displayed on the display 106, and by seeing this 
object viewpoint AR image, the spectator can see the overview of AR game and can grasp the 
situation of AR game. 

[0044]Here, it can get with above-mentioned one and the image-coordinates value of the peak 
of each polygon which constitutes the virtual body 102 seen from each viewpoint (HMD107 r 
camera 103) can be computed by the coordinate transformation matrix M1 or M2 from object 
frame to an image-coordinates system. The generation process of these coordinate 
transformation matrices is explained using the block diagram shown in dr awing 3 . 
[0045]Based on the information on the positional attitude of the virtual body 102, the coordinate 
transformation matrix Mm from object frame to a world coordinate system is computed. Based 
on HMD107, the position of each viewpoint included in each camera parameter of the camera 
103, and the information on a posture, coordinate-transformation-matrix Mc1 from a world 
coordinate system to a camera coordinate system and Mc2 are computed, respectively. Based 
on the information, including each field angle and focal distance, distortion, etc., included in each 
camera parameter of HMD107 and the camera 103, coordinate-transformation-matrix Md1 which 
performs transparent transformation from a camera coordinate system to an image-coordinates 
system, and Md2 are computed, respectively. Each above coordinate transformation matrix is 
computed in the subjectivity viewpoint video generation part 212 or the object viewpoint video 
generation part 202. 

[0046]The transformation matrix M1 from object frame to the image-coordinates system of 
HMD107 is computed by M1=Md1Mc1Mm. On the other hand, the transformation matrix M2 from 
object frame to the image-coordinates system of the camera 103 is computed by 
M2=Md2Mc2Mm. The above operation is performed by the subjectivity viewpoint video 
generation part 212 or the object viewpoint video generation part 202. 

[0047]Here, the composition and its operation inside the AR game device 104 are explained using 
drawing 7 in which the block diagram inside the AR game device 104 was shown. 
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[0048]701 is CPU and executes the program code loaded to RAM703. It also has the area which 
saves data temporarily during execution of a program. 

[0049]702 is ROM and setting out after starting and a starting program code are stored at the 
time of starting of the AR game device 104. The character code etc. which are used when 
displaying the score etc. which are outputted into AR game HMD107 on the display screen 501 
are stored. 

[0050]703 is RAM and stores the program code of AR game loaded from the floppy disk which is 
an external storage, CD?ROM F etc., the polygon which constitutes the virtual body 102, the data 
of a texture, etc. The position of the camera 103 and the data of a posture are also stored. The 
game condition Management Department 201 has managed with reference to the data of the 
polygon (texture) which constitutes the virtual body 102 stored in RAM703. 
[0051]704 is an interface part (following, I/F), and when it connects with the AR game device 
104 and an external device, it is used. HMD107, the camera 103, and the display 106 are 
altogether connected to the AR game device 104 via this I/F704. 

[0052]705 is a final controlling element, is constituted by pointing devices, such as a keyboard 
and a mouse, and can input the setting command to the peripheral equipment linked to the input 
and the AR game device 104 of a setting command of the AR game device 104. 
[0053] 706 is a bus which connects above-mentioned each part. The game condition Management 
Department 201, the object viewpoint video generation part 202, the object viewpoint image 
compositing section 203, the subjectivity viewpoint video generation part 212, the subjectivity 
viewpoint image compositing section 213, and the camera parameter instrumentation department 
204 are connected with this bus 706, It is controlled by the program code which gets with 
above-mentioned one and is loaded to RAM703 via the bus 706. 

[0054].When CPU701 executes the program code loaded to RAM703. The processing above- 
mentioned [ each part of the game condition Management Department 201, the object viewpoint 
video generation part 202, the object viewpoint image compositing section 203, the subjectivity 
viewpoint video generation part 212, the subjectivity viewpoint image compositing section 213, 
and the camera parameter instrumentation department 204 ] is performed, A subjectivity 
viewpoint AR image and an object viewpoint AR image are outputted to HMD107 and the display 
106, respectively. The flow chart of this program code is shown in drawing 8, and is explained. 
[0055]In Step S801, the camera parameter instrumentation department 204 updates the 
information on the view position posture of HMD 107. 

[0056]In Step S802, the game condition Management Department 201 updates the state 
(information about drawing of the virtual body 102) of a game. 

[0057]In Step S803, the object viewpoint video generation part 202 and the subjectivity 
viewpoint video generation part 212 generate an object viewpoint virtual image and a subjectivity 
viewpoint virtual image, respectively. 

[0058]In Step S804, the object viewpoint image compositing section 203 and the subjectivity 
viewpoint image compositing section 213 input the on-the-spot photo image picturized with 
HMD107 and the camera 103, respectively. 

[0059]In Step S805, the object viewpoint image compositing section 203 and the subjectivity 
viewpoint image compositing section 213 generate an object viewpoint AR image and a 
subjectivity viewpoint AR image, respectively. 

[0060]In Step S806, the object viewpoint image compositing section 203 and the subjectivity 
viewpoint image compositing section 213 output an object viewpoint AR image to the display 
106, and output a subjectivity viewpoint AR image to HMD 107, respectively. 
[0061]The above processing is performed until AR game is completed. 

[0062]In AR game using HMD107 of the video see-through type, the composition of the device 
of a more than, a mixed reality presenting method, and a program code can show third parties 
other than player 101 an object viewpoint AR image. 

[0063][A 2nd embodiment] In a 1st embodiment, although HMD107 was HMD of the video see- 
through type, even if HMD107 is an optical see-through type, the player 101 can perform AR 
game. 

[0064] Drawing 1 3 is a figure showing HMD1301 of an optical see-through type. HMD1301 in the 
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figure is a schematic illustration, and HMD1301 are not the size shown in the figure, and the 
thing restricted to shape. 

[0065] 1301 is HMD of an optical see-through type, and 1302 is an eye of the player 101. 
[0066]Only the image (subjectivity viewpoint virtual image) of the virtual body 102 is displayed on 
the display screen 501. On the other hand, the image of real space is seen from the position of 
the eye 1302, and is in sight in the inner part of the display screen 501. Therefore, the player 
101 can see in piles the real space which sees from the image of the virtual body 102, and the 
position of the eye 1302, and is visible in the inner part of the display screen 501 by seeing the 
display screen 501. 

[0067]The block diagram showing the flow of processing of this embodiment is shown in dr awing 
14. 

[0068]It deals in the image outputted to HMD1301 from the AR game device 104 with that of 
above-mentioned explanation, and it is only a subjectivity viewpoint virtual image. Therefore, 
HMD1301 and the subjectivity viewpoint video generation part 212 are electrically connected via 
a cable, and the subjectivity viewpoint virtual image based on the position of HMD1301 
generated in the subjectivity viewpoint video generation part 212 and a posture is sent to 
HMD1301 from the subjectivity viewpoint video generation part 212. In this embodiment, the 
subjectivity viewpoint image compositing section 213 in a 1st embodiment does not exist. 
[0069]It is the method of dealing with that of the explanation in a 1st embodiment about the 
generation method of an object viewpoint AR image. 

[0070]The block diagram inside the AR game device 104 in this embodiment removes the 
subjectivity viewpoint image compositing section 213 from the composition of d rawing 7 . 
[0071]The flow chart of this embodiment is a flow chart which rewrote Step S804 of drawing_8, 
Step S805, and Step S806 as follows. 

[0072]That is, in generation of the subjectivity viewpoint AR image (only image of the virtual 
body 102 on this embodiment and based on the position of HMD1301, and a posture) displayed 
on HMD1301, Step S804 and Step S805 become unnecessary. That is, in Step S804, the object 
viewpoint image compositing section 203 inputs the on-the-spot photo image picturized with the 
camera 103. In Step S805, the object viewpoint image compositing section 203 generates an 
object viewpoint AR image. In Step S806, the object viewpoint image compositing section 203 
outputs an object viewpoint AR image to the display 106, and the subjectivity viewpoint video 
generation part 212 outputs a subjectivity viewpoint virtual image to HMD1301, 
[0073]As a result of changing drawing 8 into getting with that of a more than, the flow chart 
which was able to be done is a flow chart in this embodiment, and this embodiment is controlled 
by the program code according to this flow chart. 

[0074]In AR game using HMD1301 of the optical see-through type, the composition of the device 
of a more than, a mixed reality presenting method, and a program code enable it to show third 
parties other than player 101 an object viewpoint AR image. 

[0075][A 3rd embodiment] The camera parameter of the camera 103 was being fixed in the 1st 
and 2 embodiment. That is, the object viewpoint AR image was an image based on the camera 
parameter of the fixed camera 103. And the data of the camera parameter of the camera 103 
was stored as a value fixed to RAM703 inside the AR game device 104, 

[0076]Then, the case where the position and posture of a viewpoint are changed into real time 
among the camera parameters of the camera 103 at the position and posture for which third 
parties other than player 101 or player 101 wish is considered. That is, the case where the 
camera parameter of the camera 103 is changed into real time is considered. HMD used in this 
embodiment uses the video see-through type used in a 1st embodiment. However, HMD which 
can be used in this embodiment was not limited to a video see-through type, and it is clear by 
the explanation in the 2nd embodiment and this embodiment that it may be an optical see- 
through type. 

[0077]It is necessary to add the measurement means which is a means to measure the position 
of the camera 103, and a posture further to a 1st embodiment to change the position of the 
camera 1 03, and a posture into real time. 

[0078]Drawing 4 is a block diagram showing the flow of the processing in this embodiment. It 
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explains flowing into processing of this embodiment using the figures below, 
[0079]Like a 1st embodiment, also in this embodiment, the camera parameter instrumentation 
department 204 carries out instrumentation of the camera parameter of HMD107, and holds the 
interna! parameter of the camera 103 as known information. That this embodiment differs from a 
1st embodiment is the point that the camera parameter instrumentation department 204 controls 
the sensor not shown in the figure with which the camera 103 was equipped, and measures the 
external parameter (the position of a viewpoint, and the information on a posture) of the camera 
103. 

[0080]When the camera 103 is located with an interface not shown in the figure and the player 
101 changes a posture, the camera 103 is located and the camera parameter instrumentation 
department 204 measures a posture. And the data (the position of the camera 103, the data of a 
posture) of the measuring result by the camera parameter instrumentation department 204 and 
the internal parameter which is known information are outputted to the object viewpoint video 
generation part 202, And the object viewpoint video generation part 202 generates an object 
viewpoint virtual image like the processing in a 1st embodiment based on the camera parameter 
of the inputted camera 103. About other processings, it is the same as that of a 1st embodiment. 

[0081]The position of the camera 103, the control program code of the interface to which a 
posture is changed, and the control program code of the camera parameter instrumentation 
department 204 are stored in RAM703 of the AR game device 104. 

[0082]The flow chart in this embodiment is the same as the flow chart of drawing_8. However, in 
Step S801, the camera parameter instrumentation department 204 updates HMD 107, the 
position of the viewpoint of the camera 103, and the information on a posture. 
[0083]Generation of an object viewpoint AR image with the camera from which it is located and 
a posture changes with the composition of the device of a more than, a mixed reality presenting 
method, and program codes is attained. 

[0084][A 4th embodiment] Only one camera is not set up in the embodiment of the 1st thru/or 
3. However, two or more positions and the object viewpoint AR image from a posture can be 
displayed on the display 106 by setting up two or more cameras and changing the camera to be 
used. 

[00853 Drawin g 9 shows the camera system in this embodiment, and explains this camera system 
hereafter using the figures. Three cameras are set up in this embodiment. 
[0086]901a, 901b, and 901c are the cameras (the camera 901a, the camera 901b, the camera 
901c) with which camera parameters, such as a position, a posture, and a field angle, were fixed. 
About a function, it is the same as the camera in the 1 st and 2 embodiment. Each camera 
parameter of three cameras has composition shown in drawing 1 2, and these three data is stored 
in RAM703 of the AR game device 104, respectively, and is managed in the camera parameter 
instrumentation department 204. 

[0087]In drawin g 1 2, 1201-1202 and 1203 are the tags to the camera 901a, the camera 901b, and 
the camera 901c (selection information), respectively, and it is used when choosing the camera 
to be used. 

[0088]902 is a transfer device of a camera and has a function as a gateway which lets the 
transfer device 902 pass and sends only the signal (on-the-spot photo image of the real space 
picturized with this camera) from a camera with the selected selector 903 to the image 
compositing section 203. 

[0089]903 has three buttons (the button A, the button B, the button C) with an above- 
mentioned function respectively corresponding to [ are a selector and ] three cameras, and 
chooses the camera which is choosing and pushing and uses one out of these buttons. The 
selector 903 outputs the selection information which shows whether which camera was chosen 
to the transfer device 902 and the camera parameter control section 204. If the button A in a 
figure is pushed, the camera 901a will push the button B and the camera 901b will push the 
button C, the camera 901c will be chosen as a camera used, respectively. 

[0090] According to the selection information of the camera inputted from the selector 903, the 
camera parameter control section 204 chooses the tag (the tag 1201, the tag 1202, the tag 
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1203) of the camera explained in ****, and outputs the camera parameter of the selected 
camera to the object viewpoint video generation part 202, For example, when using the camera 
901b, the camera parameter of the camera 901b which the tag 1202 holds is outputted to the 
object viewpoint video generation part 202 by pushing the button B. 
[0091]The flow chart of processing of this embodiment is shown in drawing 10 . 
[0092]ln Step S1001 r it is judged whether the button of the selector 903 is pushed. This 
processing is continued until a button is pushed. 

[0093]In Step S1002 r Step S1003, and Step S1004, it judges which button was pushed. 
Supposing the button B is pushed here, it will progress to Step S1006 from Step S1003. 
[0094]ln Step S1005, Step S1006, and Step S1007, the on-the-spot photo image picturized with 
the camera corresponding to the pushed button is sent to the object viewpoint image 
compositing section 203 via the transfer device 902. The selection information of the camera 
corresponding to the pushed button is sent to the camera parameter instrumentation 
department 204. 

[0095]The program code according to the above flow chart is stored in the memory not shown in 
the figure [ inside the transfer device 902 ]. 

[0096]Although the camera parameter of three cameras was being fixed in this embodiment 
when the camera parameter of each camera changes to real time, the composition of an above- 
mentioned device, a mixed reality presenting method, and a program code can be applied. In that 
case, in a 3rd embodiment the camera system of this embodiment is solvable by connecting with 
the AR game device 104 instead of the camera 103. 

[0097]Two or more object viewpoint AR images depended on two or more cameras can be 
displayed on the display 106 with the composition of the device of a more than, a mixed reality 
presenting method, and a program code. 

[0098][A 5th embodiment] In the embodiment of the 1st thru/or 4, an object viewpoint AR image 
is outputting to the display 106, and presentation ****** may show this object viewpoint AR 
image to third parties other than a player via this display 106 at the player 101. That is, the 
viewing area 601 as shown in drawing 6 is formed in the display screen 501 of HMD (video see- 
through or optical see-through which type may be used) with which the player 101 has equipped, 
and an object viewpoint AR image is displayed there. In the figure, the image currently displayed 
on this display screen 501 is hereafter called a compound image. 

[0099]In order to generate this compound image, the viewing area 601 is set as the display 
screen 501, and, in addition to the program code according to the flow chart shown in drawin g 8, 
the program code which writes an object viewpoint AR image in that viewing area 601 is stored 
in RAM703 of the AR game device 104. As a result, a compound image can be displayed on the 
display screen 501 by executing this program code. 

[0100]The flow chart of the program code which writes an object viewpoint AR image in the 
viewing area 601 explained in **** is shown in drawing 1 1 . 

[0101]In Step S1101, it is judged whether the viewing area 601 is formed. Selection of whether 
to form the viewing area 601 is realizable by forming the switch of this selection in the operation 
device not shown in the figure used when the player 101 performs AR game. The input of the 
command of whether to form the viewing area 601 can also perform from the final controlling 
element 705. 

[0102]The display position of the viewing area 601 is inputted in Step S1102. This input is 
inputted from the final controlling element 705. Or the player 101 may input using said operation 
device, 

[0103]The size of the viewing area 601 is inputted in Step S1 103. This input is inputted from the 
final controlling element 705. Or the player 101 may input using said operation device. 
[0104]In Step S1 104, the viewing area 601 by which it was decided in Step S1102 and Step 
S1 103 that setting out would be the display screen 501 is formed. 

[0105]In Step S1 105, the object viewpoint AR image generated by the viewing area 601 by the 
object viewpoint image compositing section 203 is drawn. As a result, a compound image is 
generable. 

[0106]This compound image may be outputted to the display 106. 
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[0107]The composition of the device of a more than, a mixed reality presenting method, and a 
program code can show the player 101 both a subjectivity viewpoint AR image and an object 
viewpoint AR image, 

[A 6th embodiment] The number of the players 102 of AR game may be [ two or more ]. In that 
case, it is necessary to provide the subjectivity viewpoint AR image from each subjectivity 
viewpoint to each player. The internal configuration of AR game device for providing draw ing 15 
with the subjectivity viewpoint AR image from the subjectivity viewpoint of each player to each 
player is shown. It is being fixed and the camera 103 makes processing about the camera 103 
and a display the same as a 1st embodiment. 

[0108]This figure shows the interna! configuration for three persons' player. Therefore, the inside 
of the AR game device 104 is equipped with HMD107A, 107B and 107C, the subjectivity 
viewpoint image compositing sections 21 3A, 213B t and 21 3C, and the subjectivity viewpoint 
video generation parts 21 2A, 21 2B, and 21 2C to each player a, b, and c of three persons. And 
three persons' player a, b, and c equips with HMD107A, and 107B and 107C, respectively. 
Although HMD used by this embodiment is a thing of a video see-through type, an optical see- 
through type may be sufficient as this HMD. In that case, it is solvable by excluding each 
subjectivity viewpoint image compositing section to each HMD. 

[0109]And if AR game starts, processing in which it deals that each HMD and each subjectivity 
viewpoint video generation part, and each subjectivity viewpoint image compositing section 
explained in a 1st embodiment will be performed, The subjectivity viewpoint AR image generated 
for [ each ] players to each player is outputted to HMD107A with which each player equips, and 
107Band107C. 

[0110]The above-mentioned composition and mixed reality presenting method of the device 
were not limited to the number of players being three, and are clearer than above-mentioned 
explanation. [ of this ] 

[01 11]A subjectivity viewpoint AR image can be provided to two or more players with the above 
composition and mixed reality presenting method of a device. 

[01 12][A 7th embodiment] At a 4th embodiment, even if two or more cameras 103 were set up, 

there was only the one display 106 which displays an object viewpoint AR image. 

[01 13] However, this display 106 was not limited to one and there may be two or more sets. That 

is, by dividing the object viewpoint AR image from each camera 103 into two or more sets of the 

displays 106, and displaying it, the object viewpoint AR image from all the cameras 103 can be 

shown to third parties other than player 101 by two or more sets of the displays 106, 

[01 14]This embodiment describes the case where two or more sets of two or more cameras 103 

and the displays 106 are set up. The internal configuration of the AR game device 104 for 

displaying the object viewpoint AR image from two or more cameras 103 on two or more sets of 

the displays 106 is shown in dr awing 1 6 . 

[01 15]In this figure, the three cameras 103A, 103B, and 103C were set up, and the displays 
106A, 106B, and 106C which display the object viewpoint AR image from each camera 103A, 
103B r and 103C are set up. In order to generate the object viewpoint AR image corresponding to 
each displays 106A, 106B, and 106C And each display 106A, The inside of the AR game device 
104 is equipped with the object viewpoint video generation parts 202A, 202B, and 202C 
corresponding to 106B and 106C, and the object viewpoint video generation parts 203A, 203B, 
and 203C. 

[01 16]And if AR game starts, processing in which it deals that each camera, each object 

viewpoint video generation part, and each object viewpoint image compositing section explained 

in a 1st embodiment will be performed, and the object viewpoint AR image from each camera 103 

will be displayed on each display 106 corresponding to each camera 103. 

[01 17]The display 106 was not limited to three sets and this of the above-mentioned 

composition and mixed reality presenting method of a device is clear by above-mentioned 

explanation. 

[01 18]AII the object viewpoint AR images from two or more cameras 103 can be shown to third 
parties other than player 101 on two or more displays 106 the above composition and mixed 
reality presenting method of a device. 
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[01 19] [An 8th embodiment] Except a game device may be sufficient as the device which 
provides the player 101 with AR image. Jam, Instead of the AR game device 104, internal 
composition is almost the same and it is clear that AR device with which the program code 
carried carried not the program code of a game but the program code for carrying out shopping 
in AR space may be used. As a result, the player 101 can peruse the goods in shopping shown by 
seeing the subjectivity viewpoint AR image from AR device, and the situation is displayed on the 
display 106 as an object viewpoint AR image. 

[0120]About generation of this subjectivity viewpoint AR image and an object viewpoint AR 
image, it is the same as that of an above-mentioned embodiment. 

[0121][A 9th embodiment] A positional attitude, a zoom rate, etc. of the camera 103 may be 
presumed from the on-the-spot photo picture inputted not from a sensor but from the camera 
103, when a camera parameter changes. In that case, the camera parameter instrumentation 
department 204 inputs an on-the-spot photo image, and presumes a camera parameter by a 
publicly known method. If the error in the case of this presumption is also taken into 
consideration, the camera parameter used eventually can also be determined by processing 
taking an average with the camera parameter by a sensor, and the camera parameter by 
presumption of the camera parameter instrumentation department 204 etc. 
[01 2 2] Embodiment] besides [the purpose of an above-mentioned embodiment the storage (or 
recording medium) which recorded the program code of the software which realizes the function 
of an embodiment mentioned above, It cannot be overemphasized that it is attained, also when a 
system or a device is supplied and the computer (or CPU and MPU) of the system or a device 
reads and executes the program code stored in the storage. In this case, the function of an 
embodiment which the program code itself read from the storage mentioned above will be 
realized, and the storage which memorized that program code will constitute an above- 
mentioned embodiment. 

[0123]By executing the program code which the computer read, Based on directions of the 
program code the function of an embodiment mentioned above is not only realized, but, It cannot 
be overemphasized that it is contained also when the function of an embodiment which the 
operating system (OS) etc. which are working on a computer performed a part or all of actual 
processing, and mentioned above by the processing is realized. 

[01 24] After the program code read from the storage was written in the memory with which the 
function expansion unit connected to the expansion card inserted in the computer or the 
computer is equipped, It cannot be overemphasized that it is contained also when the function of 
an embodiment which performed a part or all of processing that CPU etc. with which the 
expansion card and function expansion unit are equipped are actual, based on directions of the 
program code, and was mentioned above by the processing is realized. 

[0125]When applying an above embodiment to the above-mentioned storage, the program code 
corresponding to the flow chart (shown in d rawing 8 , dr awing 10 , or drawing 1 1 ) explained 
previously will be stored in the storage. 

[Effect of the Invention]The image of AR game by the subjectivity viewpoint of a player is 
generated, and there is an effect which generates the overview of AR game seen from the object 
viewpoint and the image of AR game from the viewpoint of a spectator's choice. 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



TECHNICAL FIELD 



[Field of the Invention]This invention relates to the mixed reality presentation system, mixed 
reality presenting method, and storage which carry out the superimposed display of the virtual 
body to real space. 

[0002]Especially a player is related with a mixed reality presentation system, a mixed reality 
presenting method, and a storage with the function which generates the mixed reality space 
currently observed from the subjectivity viewpoint as an image seen from the object viewpoint. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



PRIOR ART 



[Description of the Prior Art]AR game device with which the conventional mixed reality (following 
AR) game system controls AR game, It is ** constituted with the head mount display 
(henceforth, HMD) with which the player which performs AR game equips, and the display which 
shows third parties other than a player the subjectivity viewpoint image of said player. 
[0004]AR game device is generating the information about drawing of a virtual body at the same 
time it controls AR game. AR game device generates the synthetic video which is an image 
which compounded the image of the virtual body generated using the information about drawing 
of this virtual body, and the image of the actual space in the subjectivity viewpoint of the player 
picturized with the camera equipped with or built in HMD. And this synthetic video is displayed 
on the display screen of HMD. The player is performing AR game, looking at this synthetic video. 
The third party who observes this AR game besides a player is shown by displaying the image of 
the subjectivity viewpoint of a player which is this synthetic video on the display prepared to 
third parties other than a player. 

[0005]In the program of some televisions, the virtual studio which constitutes the set of the 
stage of a program with a virtual body is used. The image of the program in this virtual studio is 
generated by compounding the on-the-spot photo image of the characters of the program 
picturized with the set-up camera, and the position of this camera and the image of the virtual 
studio which is visible with a posture. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

LThis document has been translated by computer So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3Jn the drawings, any words are not translated. 



EFFECT OF THE INVENTION 



[Effect of the InventionjThe image of AR game by the subjectivity viewpoint of a player is 
generated, and there is an effect which generates the overview of AR game seen from the object 
viewpoint and the image of AR game from the viewpoint of a spectator's choice. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1 This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



TECHNICAL PROBLEM 



[Problem(s) to be Solved by the Inventionjln the conventional AR (mixed reality) game system, 
third parties (spectator) other than a player can see only the image (mainly subjectivity 
viewpoint of a player) generated for the player Therefore, images of AR game by the viewpoint 
from other than the subjectivity viewpoint of a player, such as an overview of AR game seen 
from the object viewpoint and an image of AR game from the viewpoint of a spectator's choice, 
could not be seen, but neither grasp of the situation of the whole AR game nor watching a game 
of AR game from the viewpoint of a spectators choice was completed. 
[0007]The image of the program using a virtual studio is not able to be based on the object 
viewpoint image of third parties other than the characters of a program, and was not able to 
generate the image by the subjectivity viewpoint of the characters of a program. 
[0008]Therefore, it aims at generating the image of AR game by the subjectivity viewpoint of a 
player, and generating the overview of AR game seen from the object viewpoint, and the image of 
AR game from the viewpoint of a spectators choice in this invention. 



[Translation done.] 
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* NOTICES * 

JPO and INP1T are not responsible for any 
damages caused by the use of this translation, 

1This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3Jn the drawings, any words are not translated. 



MEANS 



[Means for Solving the Problem]In order to attain the purpose of this invention, the mixed reality 
presentation system according to claim 1 which is a suitable embodiment of this invention is 
provided with the following composition. Namely, a mixed reality presenting means which is a 
mixed reality presentation system which carries out the superimposed display of the virtual body 
to real space, and carries out the superimposed display of said virtual body seen from said 
subjectivity view position of said player to said real space seen from a subjectivity view position 
of a player, An object viewpoint image imaging means which picturizes an object viewpoint image 
of said real space seen from an object view position, It has an object viewpoint video 
compositing means which compounds a mixed reality image seen from said object viewpoint from 
an object viewpoint video generating means which generates an object viewpoint image of said 
virtual body seen from said object view position, an object viewpoint image of said real space, 
and an object viewpoint image of said virtual body. 

[0010] As a result, an image which looked at signs that a player was operating a virtual body in 
mixed reality space, from an object view position can be generated, and third parties other than a 
player can be shown. 

[001 1]A mixed reality presentation system which is a suitable embodiment of this invention is 
provided with the feature of the following according to claim 2. Namely, a subjectivity viewpoint 
image imaging means in which said mixed reality presenting means picturizes further a 
subjectivity viewpoint image of said real space seen from said subjectivity view position of said 
player, A subjectivity viewpoint video generating means which generates a subjectivity viewpoint 
image of said virtual body seen from said subjectivity view position of said player, It has a 
subjectivity viewpoint video compositing means which compounds a mixed reality image seen 
from said subjectivity viewpoint from a subjectivity viewpoint image of said real space, and a 
subjectivity viewpoint image of said virtual body, and a presenting means which shows said player 
a mixed reality image seen from said subjectivity view position, 

[001 2]A mixed reality presentation system which is a suitable embodiment of this invention is 
provided with the feature of the following according to claim 3. Namely, in [ said mixed reality 
presenting means penetrates a subjectivity viewpoint video generating means which generates 
further a subjectivity viewpoint image of said virtual body seen from said subjectivity view 
position of said player, and said real space, and ] a display surface top which can be viewed, It 
has a presenting means which shows said player a subjectivity viewpoint image of said virtual 
body. 

[001 3] A mixed reality presentation system which is a suitable embodiment of this invention is 
provided with the feature of the following according to claim 4. That is, it has further an 
information creating means which generates information about drawing of said virtual body, and 
both said object viewpoint video generating means and said subjectivity viewpoint video 
generating means generate an image of said virtual body using information about drawing of said 
virtual body. 

[001 4] A mixed reality presentation system which is a suitable embodiment of this invention is 
provided with the feature of the following according to claim 5. That is, said information creating 
means generates information on appearance of said virtual body, and information on a positional 
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attitude of a virtual body as information about drawing of said virtual body. 

[001 5]A mixed reality presentation system which is a suitable embodiment of this invention is 

provided with the feature of the following according to claim 6. That is, a parameter of said 

object viewpoint image imaging means is known, and said object viewpoint video generating 

means generates an object viewpoint image of said virtual body according to said known 

parameter. 

[001 6]A mixed reality presentation system which is a suitable embodiment of this invention is 
provided with the feature of the following according to claim 7. Namely, a part of parameter of 
said object viewpoint image imaging means is variable, have further a measurement means which 
measures change of said parameter, and said object viewpoint video generating means, 
According to said parameter measured by said measurement means, an object viewpoint image 
of said virtual body is generated. 

[0017]When a parameter of an object viewpoint image imaging means is changed, an object 
viewpoint video generating means inputs a parameter from a measurement means, and generates 
an object viewpoint image according to it 

[001 8]A mixed reality presentation system which is a suitable embodiment of this invention is 
provided with the feature of the following according to claim 8. That is, said parameter of said 
object viewpoint image imaging means which said measurement means measures contains one of 
a positional attitude of a viewpoint, and rates of a zoom ratio at least. 

[0019]An object viewpoint image of a virtual body is generated according to a camera parameter 
(an external parameter (positional attitude of a viewpoint), and an internal parameter (a rate of a 
zoom ratio, an aspect ratio, an optical axis center position, a distortion rate)) of an object 
viewpoint image imaging means. It is preferred for a camera parameter which a measurement 
means measures that all things dynamically changed among these parameters are included. 
[0020] 

[Embodiment of the Invention] According to an accompanying drawing, the suitable embodiment 
which applied this invention is described in detail below. 

[0021 ][A 1st embodiment] This embodiment shows the mixed reality presentation system and 
mixed reality presenting method which show third parties (the following, spectator) other than a 
player signs that the player is performing AR game in AR (mixed reality) space as one of the 
suitable embodiments of this invention, 
[0022]This embodiment is shown in dra wing 1 . 

[0023] 101 is the player which is performing AR (mixed reality) game, and has equipped the head 
with the head mount display (following, HMD) 107. HMD107 in this embodiment is a video see- 
through type. About HMD of a bidet see-through type, explanation here is omitted for publicly 
known art. 

[0024] 102 is a virtual body and is a character which appears in AR game. This virtual body 102 is 
a three-dimensional virtual body constituted by the polygon. The virtual body 102 is generated 
by the method of mentioning later with the AR game device 104. And the player 101 can see the 
image (henceforth, AR image) superimposed on the virtual body 102 by real space from a 
subjectivity viewpoint by equipping a head with HMD 107, and, as a result, can perform AR game. 
The component of the virtual body 102 is not having restricted to the polygon. Otherwise, it may 
be constituted by spatial light beam data. 

[0025] 103 is the object viewpoint image imaging camera (the following, camera) 103, and is a 
camera which picturizes signs that the player 101 is performing AR game. This camera 103 is 
being fixed to the position and the posture in this embodiment The image (following and object 
viewpoint on-the-spot photo image) picturized with the camera 103 is sent to the AR game 
device 104 via the cable in a figure. The picturized image is an on-the-spot photo image, and the 
virtual body 102 is not picturized by this on-the-spot photo image. 

[0026] 104 is AR game device and is generating AR image outputted to control of AR game, 
generation of the image of the virtual body 102, HMD107, and the display 106. Here, the image 
which the AR game device 104 outputs to HMD 107 is an AR image (following and subjectivity 
viewpoint AR image) which compounded the on-the-spot photo image (following and subjectivity 
viewpoint on-the-spot photo image) of the real space inputted from HMD107, and the image 
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(following and subjectivity viewpoint virtual image) of the virtual body 102 seen from the 
subjectivity viewpoint. On the other hand, the image which the AR game device 104 outputs to 
the display 106 is an AR image (following and object viewpoint AR image) which compounded the 
image (following and object viewpoint virtual image) of the virtual body 102 seen from the object 
viewpoint and the object viewpoint on-the-spot photo image picturized with the camera 103. 
[0027]105 is a stand of the realistic object used in AR game. 

[0028] 106 is a display, is displaying the object viewpoint AR image which got with above- 
mentioned one and was generated by the AR game device 104, and shows third parties other 
than player 101 signs that the player 101 is performing AR game. 

[0029]107 is above-mentioned HMD and displays the subjectivity viewpoint AR image generated 
by the AR game device 104 on the display screen 501 shown in drawin g 5 of HMD107. HMD107 
picturizes the subjectivity viewpoint on-the-spot photo image seen from the subjectivity 
viewpoint of the player 101. The picturized image is sent to the AR game device 104 via the 
cable in a figure. 

[0030]It explains using drawin g 2 in which the flow of this processing was shown about flowing 
into processing based on the above composition until the AR game device 104 displayed a 
subjectivity viewpoint AR image and an object viewpoint AR image on HMD107 and the display 
106, respectively. 

[0031]201 is the game condition Management Department built in the AR game device 104, and 
manages the states (the information about drawing of the virtual body 102, the score of the 
player 1 01 , AR game round number, etc.) of AR game. Here, the information about drawing of the 
virtual body 102 is constituted by the information on the positional attitude of the virtual body 
102 in the information and world coordinate system of appearance of the virtual body 102. 
[0032]The information on the appearance of the virtual body 102 is information about the 
polygon which constitutes the virtual body 102, and is the number of polygons, a coordinate 
value of each polygon, a color of each polygon, etc. When texture mapping is given to the virtual 
body 102, the size of a texture, the file name of a texture, etc. are contained in the information 
on the appearance of the virtual body 102. 

[0033]204 is a camera parameter instrumentation department and is performing 
measurement/management of the camera parameter which is a parameter of HMD 107 and the 
camera 103. The position of a viewpoint and the information on a posture which are externa! 
parameters, and the information, including a field angle, a focal distance, distortion, etc., which is 
internal parameters are included in the camera parameter which the camera parameter 
instrumentation department 204 manages. The camera parameter instrumentation department 
204 controls the sensor not shown in the figure of HMD107 at the same time it holds the internal 
parameter of HMD 107 as known information, The external parameter (the position of a viewpoint 
and the information on a posture) of HMD107 was measured, and the camera parameter of 
HMD107 is managed. The camera parameter instrumentation department 204 has managed the 
camera parameter of the camera 103 as known information. 

[0034]202 is the object viewpoint video generation part built in the AR game device 104, and 

generates an object viewpoint virtual image based on the information about drawing of the virtual 

body 102 inputted from the game condition Management Department 201. 

[0035]203 is the object viewpoint image compositing section built in the AR game device 104, 

compounds the object viewpoint virtual image generated by the object viewpoint video 

generation part 202 and the object viewpoint onH:he-spot photo image inputted from the camera 

103, and generates an object viewpoint AR image. 

[0036] First, generation of a subjectivity viewpoint image is explained. 

[0037]The game condition Management Department 201 updates the information about drawing 
of the virtual body 102 if needed, and outputs update information to the subjectivity viewpoint 
video generation part 212. The subjectivity viewpoint video generation part 212 generates the 
image of the virtual body 102 based on the camera parameter of HMD107 inputted as the 
information about drawing of the virtual body 102 inputted from the game condition Management 
Department 201 from the camera parameter instrumentation department 204. Specifically, the 
coordinate transformation matrix M1 from object frame to the image-coordinates system of 
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HMD107 is computed based on the method of mentioning later from the information on the 
positional attitude of the virtual body 102, and the camera parameter of HMD107. By hanging the 
coordinate transformation matrix M1 computed to the apex-coordinates value in the object 
frame of each polygon which constitutes the virtual body 102, The image-coordinates value of 
the peak of each polygon which constitutes the virtual body 102 is computed, and the image 
(subjectivity viewpoint virtual image) of the virtual body 102 seen from the view position posture 
of HMD107 is generated. And the generated subjectivity viewpoint virtual image is outputted to 
the subjectivity viewpoint image compositing section 213. 

[0038]The subjectivity viewpoint on-the-spot photo image which the player 101 looks at via 
HMD107 is inputted into the subjectivity viewpoint image compositing section 213 from HMD107. 
And the subjectivity viewpoint image compositing section 213 generates the subjectivity 
viewpoint AR image which is synthetic video with the subjectivity viewpoint virtual image 
inputted from this subjectivity viewpoint on-the-spot photo image and the subjectivity viewpoint 
video generation part 212, and outputs this subjectivity viewpoint AR image to HMD107, A 
subjectivity viewpoint AR image is displayed on the display screen 501 of HMD107, and the 
player 101 performs AR game, seeing this subjectivity viewpoint AR image. 
[0039] Next, generation of an object viewpoint image is explained. 

[0040]The game condition Management Department 201 updates the information about drawing 
of the virtual body 102 if needed, and outputs update information to the object viewpoint video 
generation part 202. 

[0041]Since it is the processing same about generation of the image of the virtual body 102 as 
the processing in generation of a subjectivity viewpoint image, explanation is omitted here. 
[0042]The object viewpoint video generation part 202 generates the image of the virtual body 
102 based on the camera parameter of the camera 103 inputted as the information about 
drawing of the virtual body 102 inputted from the game condition Management Department 201 
from the camera parameter instrumentation department 204. Specifically, the coordinate- 
system-transformation procession M2 from object frame to the image-coordinates system of 
the camera 103 is computed based on the method of mentioning later from the information on 
the positional attitude of the virtual body 102, and the camera parameter of the camera 103. By 
hanging the coordinate transformation matrix M2 computed to the apex-coordinates value in the 
object frame of each polygon which constitutes the virtual body 102. The image-coordinates 
value of the peak of each polygon which constitutes the virtual body 102 is computed, and the 
image (object viewpoint virtual image) of the virtual body 103 seen from the view position 
posture of the camera 103 is generated. And the generated object viewpoint virtual image is 
outputted to the object viewpoint image compositing section 203. 

[0043]An object viewpoint on-the-spot photo image is inputted into the object viewpoint image 
compositing section 203 from the camera 103. And the object viewpoint image compositing 
section 203 generates the object viewpoint AR image which is synthetic video with the object 
viewpoint virtual image inputted from this object viewpoint on-the-spot photo image and the 
object viewpoint video generation part 202, and outputs this object viewpoint AR image to the 
display 106. This object viewpoint AR image is displayed on the display 106, and by seeing this 
object viewpoint AR image, the spectator can see the overview of AR game and can grasp the 
situation of AR game. 

[0044]Here, it can get with above-mentioned one and the image-coordinates value of the peak 
of each polygon which constitutes the virtual body 102 seen from each viewpoint (HMD 107, 
camera 103) can be computed by the coordinate transformation matrix M1 or M2 from object 
frame to an image-coordinates system. The generation process of these coordinate 
transformation matrices is explained using the block diagram shown in drawing 3 . 
[0045]Based on the information on the positional attitude of the virtual body 102, the coordinate 
transformation matrix Mm from object frame to a world coordinate system is computed. Based 
on HMD107, the position of each viewpoint included in each camera parameter of the camera 
103, and the information on a posture, coordinate-transformation-matrix Mc1 from a world 
coordinate system to a camera coordinate system and Mc2 are computed, respectively. Based 
on the information, including each field angle and focal distance, distortion, etc., included in each 
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camera parameter of HMD107 and the camera 103, coordinate-transformation-matrix Mdl which 
performs transparent transformation from a camera coordinate system to an image-coordinates 
system, and Md2 are computed, respectively. Each above coordinate transformation matrix is 
computed in the subjectivity viewpoint video generation part 212 or the object viewpoint video 
generation part 202. 

[0046]The transformation matrix M1 from object frame to the image-coordinates system of 
HMD107 is computed by M1=Md1Mc1Mm. On the other hand, the transformation matrix M2 from 
object frame to the image-coordinates system of the camera 103 is computed by 
M2=Md2Mc2Mm. The above operation is performed by the subjectivity viewpoint video 
generation part 212 or the object viewpoint video generation part 202. 

[0047]Here, the composition and its operation inside the AR game device 104 are explained using 
dr awing 7 in which the block diagram inside the AR game device 104 was shown. 
[0048]701 is CPU and executes the program code loaded to RAM703. It aiso has the area which 
saves data temporarily during execution of a program. 

[0049]702 is ROM and setting out after starting and a starting program code are stored at the 
time of starting of the AR game device 104. The character code etc. which are used when 
displaying the score etc. which are outputted into AR game HMD107 on the display screen 501 
are stored. 

[0050]703 is RAM and stores the program code of AR game loaded from the floppy disk which is 
an external storage, CD?ROM t etc., the polygon which constitutes the virtual body 102, the data 
of a texture, etc. The position of the camera 1 03 and the data of a posture are also stored. The 
game condition Management Department 201 has managed with reference to the data of the 
polygon (texture) which constitutes the virtual body 102 stored in RAM703. 
[0051]704 is an interface part (following, I/F), and when it connects with the AR game device 
104 and an external device, it is used. HMD107, the camera 103, and the display 106 are 
altogether connected to the AR game device 104 via this I/F704. 

[0052]705 is a final controlling element, is constituted by pointing devices, such as a keyboard 
and a mouse, and can input the setting command to the peripheral equipment linked to the input 
and the AR game device 104 of a setting command of the AR game device 104. 
[0053]706 is a bus which connects above-mentioned each part. The game condition Management 
Department 201, the object viewpoint video generation part 202, the object viewpoint image 
compositing section 203, the subjectivity viewpoint video generation part 212, the subjectivity 
viewpoint image compositing section 213, and the camera parameter instrumentation department 
204 are connected with this bus 706, It is controlled by the program code which gets with 
above-mentioned one and is loaded to RAM703 via the bus 706. 

[0054].When CPU701 executes the program code loaded to RAM703. The processing above- 
mentioned [ each part of the game condition Management Department 201, the object viewpoint 
video generation part 202, the object viewpoint image compositing section 203, the subjectivity 
viewpoint video generation part 212, the subjectivity viewpoint image compositing section 213, 
and the camera parameter instrumentation department 204 ] is performed, A subjectivity 
viewpoint AR image and an object viewpoint AR image are outputted to HMD107 and the display 
106, respectively. The flow chart of this program code is shown in dra wing 8 , and is explained. 
[0055]In Step S801, the camera parameter instrumentation department 204 updates the 
information on the view position posture of HMD 107. 

[0056]In Step S802, the game condition Management Department 201 updates the state 
(information about drawing of the virtual body 102) of a game. 

[0057]In Step S803, the object viewpoint video generation part 202 and the subjectivity 
viewpoint video generation part 212 generate an object viewpoint virtual image and a subjectivity 
viewpoint virtual image, respectively. 

[0058]In Step S804, the object viewpoint image compositing section 203 and the subjectivity 
viewpoint image compositing section 213 input the on-the-spot photo image picturized with 
HMD107 and the camera 103, respectively. 

[0059]In Step S805, the object viewpoint image compositing section 203 and the subjectivity 
viewpoint image compositing section 213 generate an object viewpoint AR image and a 
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subjectivity viewpoint AR image, respectively. 

[0060]In Step S806, the object viewpoint image compositing section 203 and the subjectivity 
viewpoint image compositing section 213 output an object viewpoint AR image to the display 
106, and output a subjectivity viewpoint AR image to HMD107, respectively. 
[0061]The above processing is performed until AR game is completed. 

[0062]In AR game using HMD107 of the video see-through type, the composition of the device 
of a more than, a mixed reality presenting method, and a program code can show third parties 
other than player 101 an object viewpoint AR image. 

[0063][A 2nd embodiment] In a 1st embodiment, although HMD107 was HMD of the video see- 
through type, even if HMD107 is an optical see-through type, the player 101 can perform AR 
game. 

[0064] Drawing 13 is a figure showing HMD1301 of an optical see-through type. HMD1301 in the 
figure is a schematic illustration, and HMD1301 are not the size shown in the figure, and the 
thing restricted to shape. 

[0065] 1301 is HMD of an optical see-through type, and 1302 is an eye of the player 101. 
[0066]Only the image (subjectivity viewpoint virtual image) of the virtual body 102 is displayed on 
the display screen 501. On the other hand, the image of real space is seen from the position of 
the eye 1302, and is in sight in the inner part of the display screen 501. Therefore, the player 
101 can see in piles the real space which sees from the image of the virtual body 102, and the 
position of the eye 1302, and is visible in the inner part of the display screen 501 by seeing the 
display screen 501. 

[0067]The block diagram showing the flow of processing of this embodiment is shown in d rawing 
14. 

[0068]It deals in the image outputted to HMD 1301 from the AR game device 104 with that of 
above-mentioned explanation, and it is only a subjectivity viewpoint virtual image. Therefore, 
HMD1301 and the subjectivity viewpoint video generation part 212 are electrically connected via 
a cable, and the subjectivity viewpoint virtual image based on the position of HMD1301 
generated in the subjectivity viewpoint video generation part 212 and a posture is sent to 
HMD1301 from the subjectivity viewpoint video generation part 212. In this embodiment, the 
subjectivity viewpoint image compositing section 213 in a 1st embodiment does not exist. 
[0069]It is the method of dealing with that of the explanation in a 1st embodiment about the 
generation method of an object viewpoint AR image, 

[0070]The block diagram inside the AR game device 104 in this embodiment removes the 
subjectivity viewpoint image compositing section 213 from the composition of drawing 7 . 
[0071]The flow chart of this embodiment is a flow chart which rewrote Step S804 of drawing 8, 
Step S805, and Step S806 as follows. 

[0072]That is, in generation of the subjectivity viewpoint AR image (only image of the virtual 
body 102 on this embodiment and based on the position of HMD1301, and a posture) displayed 
on HMD1301, Step S804 and Step S805 become unnecessary. That is, in Step S804, the object 
viewpoint image compositing section 203 inputs the on-the-spot photo image picturized with the 
camera 103. In Step S805, the object viewpoint image compositing section 203 generates an 
object viewpoint AR image. In Step S806, the object viewpoint image compositing section 203 
outputs an object viewpoint AR image to the display 106, and the subjectivity viewpoint video 
generation part 212 outputs a subjectivity viewpoint virtual image to HMD1301. 
[0073]As a result of changing drawi ng 8 into getting with that of a more than, the flow chart 
which was able to be done is a flow chart in this embodiment, and this embodiment is controlled 
by the program code according to this flow chart. 

[0074]ln AR game using HMD1301 of the optical see-through type, the composition of the device 
of a more than, a mixed reality presenting method, and a program code enable it to show third 
parties other than player 101 an object viewpoint AR image. 

[0075][A 3rd embodiment] The camera parameter of the camera 103 was being fixed in the 1st 
and 2 embodiment. That is, the object viewpoint AR image was an image based on the camera 
parameter of the fixed camera 103. And the data of the camera parameter of the camera 103 
was stored as a value fixed to RAM703 inside the AR game device 104. 
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[0076]Then, the case where the positfon and posture of a viewpoint are changed into real time 
among the camera parameters of the camera 103 at the position and posture for which third 
parties other than player 101 or player 101 wish is considered. That is, the case where the 
camera parameter of the camera 103 is changed into real time is considered. HMD used in this 
embodiment uses the video see-through type used in a 1 st embodiment However, HMD which 
can be used in this embodiment was not limited to a video see-through type, and it is clear by 
the explanation in the 2nd embodiment and this embodiment that it may be an optical see- 
through type. 

[0077]It is necessary to add the measurement means which is a means to measure the position 
of the camera 103, and a posture further to a 1st embodiment to change the position of the 
camera 103, and a posture into real time. 

[0078] Drawing 4 is a block diagram showing the flow of the processing in this embodiment. It 
explains flowing into processing of this embodiment using the figures below. 
[0079]Like a 1st embodiment, also in this embodiment, the camera parameter instrumentation 
department 204 carries out instrumentation of the camera parameter of HMD107 t and holds the 
interna! parameter of the camera 103 as known information. That this embodiment differs from a 
1st embodiment is the point that the camera parameter instrumentation department 204 controls 
the sensor not shown in the figure with which the camera 103 was equipped, and measures the 
external parameter (the position of a viewpoint, and the information on a posture) of the camera 
103. 

[0080]When the camera 103 is located with an interface not shown in the figure and the player 
101 changes a posture, the camera 103 is located and the camera parameter instrumentation 
department 204 measures a posture. And the data (the position of the camera 103, the data of a 
posture) of the measuring result by the camera parameter instrumentation department 204 and 
the internal parameter which is known information are outputted to the object viewpoint video 
generation part 202. And the object viewpoint video generation part 202 generates an object 
viewpoint virtual image like the processing in a 1st embodiment based on the camera parameter 
of the inputted camera 103. About other processings, it is the same as that of a 1st embodiment. 

[0081]The position of the camera 103, the control program code of the interface to which a 
posture is changed, and the control program code of the camera parameter instrumentation 
department 204 are stored in RAM703 of the AR game device 104. 

[0082]The flow chart in this embodiment is the same as the flow chart of drawin g 8. However, in 
Step S801, the camera parameter instrumentation department 204 updates HMD107, the 
position of the viewpoint of the camera 103, and the information on a posture. 
[0083] Generation of an object viewpoint AR image with the camera from which it is located and 
a posture changes with the composition of the device of a more than, a mixed reality presenting 
method, and program codes is attained. 

[0084][A 4th embodiment] Only one camera is not set up in the embodiment of the 1st thru/or 
3. However, two or more positions and the object viewpoint AR image from a posture can be 
displayed on the display 106 by setting up two or more cameras and changing the camera to be 
used. 

[0085]Drawing 9 shows the camera system in this embodiment and explains this camera system 
hereafter using the figures. Three cameras are set up in this embodiment. 
[00863901a, 901b, and 901c are the cameras (the camera 901a, the camera 901b, the camera 
901c) with which camera parameters, such as a position, a posture, and a field angle, were fixed. 
About a function, it is the same as the camera in the 1st and 2 embodiment. Each camera 
parameter of three cameras has composition shown in drawing 12, and these three data is stored 
in RAM703 of the AR game device 104, respectively, and is managed in the camera parameter 
instrumentation department 204. 

[0087]In drawin g 12, 1201-1202 and 1203 are the tags to the camera 901a, the camera 901b, and 
the camera 901c (selection information), respectively, and it is used when choosing the camera 
to be used. 

[0088]902 is a transfer device of a camera and has a function as a gateway which lets the 
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transfer device 902 pass and sends only the signal (on-the-spot photo image of the real space 
picturized with this camera) from a camera with the selected selector 903 to the image 
compositing section 203. 

[0089]903 has three buttons (the button A, the button B, the button C) with an above- 
mentioned function respectively corresponding to [ are a selector and ] three cameras, and 
chooses the camera which is choosing and pushing and uses one out of these buttons. The 
selector 903 outputs the selection information which shows whether which camera was chosen 
to the transfer device 902 and the camera parameter control section 204. If the button A in a 
figure is pushed, the camera 901a will push the button B and the camera 901b will push the 
button C, the camera 901c will be chosen as a camera used, respectively, 

[0090]According to the selection information of the camera inputted from the selector 903, the 
camera parameter control section 204 chooses the tag (the tag 1201, the tag 1202, the tag 
1203) of the camera explained in ****, and outputs the camera parameter of the selected 
camera to the object viewpoint video generation part 202. For example, when using the camera 
901b, the camera parameter of the camera 901b which the tag 1202 holds is outputted to the 
object viewpoint video generation part 202 by pushing the button B. 
[0091]The flow chart of processing of this embodiment is shown in d ra wing 10. 
[0092]In Step S1001, it is judged whether the button of the selector 903 is pushed. This 
processing is continued until a button is pushed. 

[0093]In Step S1002, Step S1003, and Step S1004, it judges which button was pushed. 
Supposing the button B is pushed here, it will progress to Step S1006 from Step S1003. 
[0094]In Step S1005, Step S1006, and Step S1007, the on-the-spot photo image picturized with 
the camera corresponding to the pushed button is sent to the object viewpoint image 
compositing section 203 via the transfer device 902. The selection information of the camera 
corresponding to the pushed button is sent to the camera parameter instrumentation 
department 204. 

[0095]The program code according to the above flow chart is stored in the memory not shown in 
the figure [ inside the transfer device 902 ]. 

[0096]Although the camera parameter of three cameras was being fixed in this embodiment, 
when the camera parameter of each camera changes to real time, the composition of an above- 
mentioned device, a mixed reality presenting method, and a program code can be applied. In that 
case, in a 3rd embodiment, the camera system of this embodiment is solvable by connecting with 
the AR game device 104 instead of the camera 103. 

[0097]Two or more object viewpoint AR images depended on two or more cameras can be 
displayed on the display 106 with the composition of the device of a more than, a mixed reality 
presenting method, and a program code. 

[0098][A 5th embodiment] In the embodiment of the 1st thru/or 4, an object viewpoint AR image 
is outputting to the display 106, and presentation ****** may show this object viewpoint AR 
image to third parties other than a player via this display 106 at the player 101. That is, the 
viewing area 601 as shown in drawing 6 is formed in the display screen 501 of HMD (video see- 
through or optical see-through which type may be used) with which the player 101 has equipped, 
and an object viewpoint AR image is displayed there. In the figure, the image currently displayed 
on this display screen 501 is hereafter called a compound image. 

[0099]In order to generate this compound image, the viewing area 601 is set as the display 
screen 501, and, in addition to the program code according to the flow chart shown in drawingJJ, 
the program code which writes an object viewpoint AR image in that viewing area 601 is stored 
in RAM703 of the AR game device 104. As a result, a compound image can be displayed on the 
display screen 501 by executing this program code. 

[0100]The flow chart of the program code which writes an object viewpoint AR image in the 
viewing area 601 explained in **** is shown in drawing 1 1 . 

[0101]In Step S1101, it is judged whether the viewing area 601 is formed. Selection of whether 
to form the viewing area 601 is realizable by forming the switch of this selection in the operation 
device not shown in the figure used when the player 101 performs AR game. The input of the 
command of whether to form the viewing area 601 can also perform from the final controlling 
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element 705. 

[0102]The display position of the viewing area 601 is inputted in Step S1102. This input is 
inputted from the final controlling element 705, Or the player 101 may input using said operation 
device. 

[0103]The size of the viewing area 601 is inputted in Step S1103. This input is inputted from the 
final controlling element 705, Or the player 101 may input using said operation device. 
[0104]In Step S1104, the viewing area 601 by which It was decided in Step S1 102 and Step 
S1 103 that setting out would be the display screen 501 is formed. 

[0105]In Step S1 105, the object viewpoint AR image generated by the viewing area 601 by the 
object viewpoint image compositing section 203 is drawn. As a result, a compound image is 
generable, 

[0106]This compound image may be outputted to the display 106. 

[0107]The composition of the device of a more than, a mixed reality presenting method, and a 
program code can show the player 101 both a subjectivity viewpoint AR image and an object 
viewpoint AR image. 

[A 6th embodiment] The number of the players 102 of AR game may be [ two or more ]. In that 
case, it is necessary to provide the subjectivity viewpoint AR image from each subjectivity 
viewpoint to each player. The internal configuration of AR game device for providing drawing 15 
with the subjectivity viewpoint AR image from the subjectivity viewpoint of each player to each 
player is shown. It is being fixed and the camera 103 makes processing about the camera 103 
and a display the same as a 1st embodiment. 

[0108]This figure shows the internal configuration for three persons 7 player. Therefore, the inside 
of the AR game device 104 is equipped with HMD107A, 107B and 107C, the subjectivity 
viewpoint image compositing sections 21 3A, 21 3B, and 213C, and the subjectivity viewpoint 
video generation parts 21 2A, 21 2B, and 21 2C to each player a, b, and c of three persons. And 
three persons 7 player a, b, and c equips with HMD107A, and 107B and 107C, respectively. 
Although HMD used by this embodiment is a thing of a video see-through type, an optical see- 
through type may be sufficient as this HMD. In that case, it is solvable by excluding each 
subjectivity viewpoint image compositing section to each HMD. 

[0109]And if AR game starts, processing in which it deals that each HMD and each subjectivity 
viewpoint video generation part, and each subjectivity viewpoint image compositing section 
explained in a 1st embodiment will be performed, The subjectivity viewpoint AR image generated 
for [ each ] players to each player is outputted to HMD107A with which each player equips, and 
107Band107C. 

[0110]The above-mentioned composition and mixed reality presenting method of the device 
were not limited to the number of players being three, and are clearer than above-mentioned 
explanation. [ of this ] 

[01 1 1]A subjectivity viewpoint AR image can be provided to two or more players with the above 
composition and mixed reality presenting method of a device. 

[01 12][A 7th embodiment] At a 4th embodiment, even if two or more cameras 103 were set up, 

there was only the one display 106 which displays an object viewpoint AR image. 

[01 13] However, this display 106 was not limited to one and there may be two or more sets. That 

is, by dividing the object viewpoint AR image from each camera 103 into two or more sets of the 

displays 106, and displaying it, the object viewpoint AR image from all the cameras 103 can be 

shown to third parties other than player 101 by two or more sets of the displays 106. 

[01 14]This embodiment describes the case where two or more sets of two or more cameras 103 

and the displays 106 are set up. The internal configuration of the AR game device 104 for 

displaying the object viewpoint AR image from two or more cameras 103 on two or more sets of 

the displays 106 is shown in drawin g 16. 

[01 15]In this figure, the three cameras 103A, 103B, and 103C were set up, and the displays 
106A, 106B, and 106C which display the object viewpoint AR image from each camera 103A, 
103B, and 103C are set up. In order to generate the object viewpoint AR image corresponding to 
each displays 106A, 106B, and 106C And each display 106A, The inside of the AR game device 
104 is equipped with the object viewpoint video generation parts 202A, 202B, and 202C 
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corresponding to 106B and 106C, and the object viewpoint video generation parts 203A T 203B, 
and 203C. 

[01 16]And if AR game starts, processing in which it deals that each camera, each object 

viewpoint video generation part, and each object viewpoint image compositing section explained 

in a 1st embodiment will be performed, and the object viewpoint AR image from each camera 103 

will be displayed on each display 106 corresponding to each camera 103. 

[01 17]The display 106 was not limited to three sets and this of the above-mentioned 

composition and mixed reality presenting method of a device is clear by above-mentioned 

explanation, 

[01 1 8] All the object viewpoint AR images from two or more cameras 103 can be shown to third 
parties other than player 101 on two or more displays 106 the above composition and mixed 
reality presenting method of a device. 

[01 19][An 8th embodiment] Except a game device may be sufficient as the device which 
provides the player 101 with AR image. Jam. Instead of the AR game device 104, internal 
composition is almost the same and it is clear that AR device with which the program code 
carried carried not the program code of a game but the program code for carrying out shopping 
in AR space may be used. As a result, the player 101 can peruse the goods in shopping shown by 
seeing the subjectivity viewpoint AR image from AR device, and the situation is displayed on the 
display 106 as an object viewpoint AR image, 

[0120]About generation of this subjectivity viewpoint AR image and an object viewpoint AR 
image, it is the same as that of an above-mentioned embodiment. 

[0121][A 9th embodiment] A positional attitude, a zoom rate, etc. of the camera 103 may be 
presumed from the on-the-spot photo picture inputted not from a sensor but from the camera 
103, when a camera parameter changes. In that case, the camera parameter instrumentation 
department 204 inputs an on-the-spot photo image, and presumes a camera parameter by a 
publicly known method. If the error in the case of this presumption is also taken into 
consideration, the camera parameter used eventually can also be determined by processing 
taking an average with the camera parameter by a sensor, and the camera parameter by 
presumption of the camera parameter instrumentation department 204 etc. 
[01 22] Embodiment] besides [the purpose of an above-mentioned embodiment the storage (or 
recording medium) which recorded the program code of the software which realizes the function 
of an embodiment mentioned above, It cannot be overemphasized that it is attained, also when a 
system or a device is supplied and the computer (or CPU and MPU) of the system or a device 
reads and executes the program code stored in the storage. In this case, the function of an 
embodiment which the program code itself read from the storage mentioned above will be 
realized, and the storage which memorized that program code will constitute an above- 
mentioned embodiment. 

[0123]By executing the program code which the computer read, Based on directions of the 
program code the function of an embodiment mentioned above is not only realized, but, It cannot 
be overemphasized that it is contained also when the function of an embodiment which the 
operating system (OS) etc. which are working on a computer performed a part or all of actual 
processing, and mentioned above by the processing is realized. 

[0124]After the program code read from the storage was written in the memory with which the 
function expansion unit connected to the expansion card inserted in the computer or the 
computer is equipped, It cannot be overemphasized that it is contained also when the function of 
an embodiment which performed a part or all of processing that CPU etc, with which the 
expansion card and function expansion unit are equipped are actual, based on directions of the 
program code, and was mentioned above by the processing is realized. 

[0125]When applying an above embodiment to the above-mentioned storage, the program code 
corresponding to the flow chart (shown in drawing_8, drawing 10 , or drawing 1 1 ) explained 
previously will be stored in the storage. 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1 This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2,**** shows the word which can not be translated. 
3Jn the drawings, any words are not translated. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1 ]lt is a figure showing a 1st embodiment. 

[Drawing 2] It is a block diagram showing the flow of processing of a 1st embodiment. 
[Drawing 3] It is a figure showing the process in which the coordinate transformation matrix which 
changes the object coordinates of a virtual body into the image coordinates seen from each 
viewpoint is generated. 

[D rawing 4] It is a block diagram showing the flow of processing of a 3rd embodiment. 

[Drawing 5] It is a figure showing the image which a player is shown in the embodiment of the 1st 

thru/or 4. 

[Drawi ng 6] It is a figure showing the image which a player is shown in a 5th embodiment. 
[Drawing 7] It is a figure showing the block diagram inside AR game device. 
[Dr awing 8] It is a flow chart which generates a subjectivity viewpoint AR image and an object 
viewpoint AR image. 

[Drawing 9] It is a figure showing the camera system used by a 4th embodiment. 
[Drawing 10jlt is a flow chart in a 4th embodiment 
[Drawing 1 1] It is a flow chart in a 5th embodiment. 

[Drawing 12]ln a 4th embodiment it is a figure showing the camera parameter of two or more 
cameras. 

[Dra wing 13] It is a figure explaining HMD of an optical see-through type. 

[Drawin g 14] It is a block diagram showing the flow of processing of a 2nd embodiment. 

[Drawing 15] It is a figure showing a 6th embodiment. 

[Drawing 16] It is a figure showing a 7th embodiment. 
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[Drawing 9] 
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WRITTEN AMENDMENT 



[Written amendment] 

[Filing datejFebruary 7, Heisei 13 (2001.2.7) 
[Amendment 1] 

[Document to be Amended]Specification 
[Item(s) to be Amended]Claim 24 
[Method of Amendment]Change 
[Proposed Amendment] 

[Claim 24]A parameter of a means to perform said object view point imag e ima ge pick-up is 
known, 

A storage which stores the program code according to claim 19 or 21, wherein a program code of 
said object viewpoint image generation process generates an object viewpoint image of said 
virtual body according to said known parameter. 
[The amendment 2] 

[Document to be Amended]Spectfication 
[Item(s) to be Amended]Claim 25 
[Method of Amendment]Change 
[Proposed Amendment] 

[Claim 25]A part of parameter of a means to perform said object viewpoint image image pick-up 
is variable, 

It has further a progra m code of a m eas urement step which measures change of said parameter, 
A storage which stores the program code according to claim 19 or 22, wherein a program code of 
said object viewpoint image generation process generates an object viewpoint image of said 
virtual body according to said parameter measured by said measuremen t step. 
[The amendment 3] 

[Document to be Amended]Specification 
[Item(s) to be Amended]Claim 26 
[Method of Amendment]Change 
[Proposed Amendment] 

[Claim 26]A storage which stores the program code according to claim 25, wherein said 
parameter of a means to perform an object viewpoint image image pick-up measured by said 
measurement ste p contains one of a positional attitude of a viewpoint, and rates of a zoom ratio 
at least. 
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